Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs are skill-demanding, time-consuming, and non-scalable to batch production. Although generative models emerge to make design automation no longer utopian, it remains non-trivial to customize designs that comply with designers' multimodal desires, i.e., constrained by background images and driven by foreground contents. In this study, we propose \textit{LayoutDETR} that inherits the high quality and realism from generative modeling, in the meanwhile reformulating content-aware requirements as a detection problem: we learn to detect in a background image the reasonable locations, scales, and spatial relations for multimodal elements in a layout. Experiments validate that our solution yields new state-of-the-art performance for layout generation on public benchmarks and on our newly-curated ads banner dataset. For practical usage, we build our solution into a graphical system that facilitates user studies. We demonstrate that our designs attract more subjective preference than baselines by significant margins. Our code, models, dataset, graphical system, and demos are available at https://github.com/salesforce/LayoutDETR.

The understanding capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt / 30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.

Image super-resolution is a common task on mobile and IoT devices, where one often needs to upscale and enhance low-resolution images and video frames. While numerous solutions have been proposed for this problem in the past, they are usually not compatible with low-power mobile NPUs having many computational and memory constraints. In this Mobile AI challenge, we address this problem and propose the participants to design an efficient quantized image super-resolution solution that can demonstrate a real-time performance on mobile NPUs. The participants were provided with the DIV2K dataset and trained INT8 models to do a high-quality 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated edge NPU capable of accelerating quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 60 FPS rate when reconstructing Full HD resolution images. A detailed description of all models developed in the challenge is provided in this paper.

The role of mobile cameras increased dramatically over the past few years, leading to more and more research in automatic image quality enhancement and RAW photo processing. In this Mobile AI challenge, the target was to develop an efficient end-to-end AI-based image signal processing (ISP) pipeline replacing the standard mobile ISPs that can run on modern smartphone GPUs using TensorFlow Lite. The participants were provided with a large-scale Fujifilm UltraISP dataset consisting of thousands of paired photos captured with a normal mobile camera sensor and a professional 102MP medium-format FujiFilm GFX100 camera. The runtime of the resulting models was evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops. The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

农作物管理，包括氮（N）受精和灌溉管理，对农作物产量，经济利润和环境产生了重大影响。尽管存在管理指南，但要在特定的种植环境和农作物中找到最佳的管理实践是挑战。先前的工作使用加强学习（RL）和作物模拟器来解决该问题，但是训练有素的政策要么具有有限的性能，要么在现实世界中不可部署。在本文中，我们提出了一种智能作物管理系统，该系统通过RL，模仿学习（IL）同时优化N受精和灌溉，并使用农业技术决策系统（DSSAT）进行了作物模拟。我们首先使用Deep RL，尤其是Deep Q-Network来培训需要从模拟器中的所有状态信息作为观测值（表示为完整观察）的管理政策。然后，我们援引IL来培训管理政策，这些政策只需要有限的国家信息，这些信息可以通过模仿以前的RL训练有素的政策在全面观察中轻松获得的国家（表示为部分观察）。我们在佛罗里达州使用玉米的案例研究进行实验，并将受过训练的政策与玉米管理指南进行比较。我们在全面观察和部分观察中训练有素的政策取得了更好的结果，从而获得更高的利润或类似的利润，而环境影响较小。此外，部分观察管理政策在使用易于使用的信息时直接在现实世界中部署。

半监督学习（SSL）通过利用大量未标记数据来增强有限标记的样品来改善模型的概括。但是，目前，流行的SSL评估协议通常受到计算机视觉（CV）任务的约束。此外，以前的工作通常从头开始训练深层神经网络，这是耗时且环境不友好的。为了解决上述问题，我们通过从简历，自然语言处理（NLP）和音频处理（AUDIO）中选择15种不同，具有挑战性和全面的任务来构建统一的SSL基准（USB），我们会系统地评估主导的SSL方法，以及开源的一个模块化和可扩展的代码库，以对这些SSL方法进行公平评估。我们进一步为简历任务提供了最新的神经模型的预训练版本，以使成本负担得起，以进行进一步调整。 USB启用对来自多个域的更多任务的单个SSL算法的评估，但成本较低。具体而言，在单个NVIDIA V100上，仅需要37个GPU天才能在USB中评估15个任务的FIXMATCH，而335 GPU天（除ImageNet以外的4个CV数据集中的279 GPU天）在使用典型协议的5个CV任务上需要进行5个CV任务。

我们考虑了决定如何最好地靶向和优先考虑现有疫苗的问题，这些疫苗可能可以保护对传染病的新变体的保护。顺序实验是一种有前途的方法。但是，由于反馈延迟以及疾病患病率的整体起伏和流动的挑战使得该任务不适用的方法。我们提出了一种可以应对这些挑战的方法，汤普森采样的方法。我们的方法涉及运行汤普森采样，每次观察事件时，都由部分可能性确定的信念更新。为了测试我们的方法，我们根据美国的Covid-19感染数据200天进行了半合成实验。

最近的基于学习的图像雨和噪声衰减的繁荣主要是由于精心设计的神经网络架构和大型标记数据集。但是，我们发现当前的图像雨和噪声去除方法导致图像的利用率低。为了减轻对大型标签数据集的依赖，我们提出了基于引入的补丁分析策略的任务驱动的图像雨和噪声（TRNR）。补丁分析策略提供了具有各种空间和统计特性的图像贴片，用于培训，并已被验证以增加图像的利用率。此外，补丁分析策略激励我们考虑学习图像雨和噪声去除任务驱动而不是数据驱动。因此，我们介绍了TRNR的N频率-K射击学习任务。每个N频率-K-Shot学习任务基于包含补丁分析策略采样的NK图像修补的微小数据集。 TRNR使神经网络能够从足够的数据以外的丰富N频率-K射击学习任务中学习。为了验证TRNR的有效性，我们建立了一个浅色多尺度残差网络（MSRESNet），具有约0.9米的参数来学习图像雨量拆卸，并使用简单的RESET与大约1.2M参数配合DNNET进行盲目高斯噪声删除，有一些图像（例如，20.0％的Rain100h培训赛车组）。实验结果表明，TRNR使MSRESNet能够从更少的图像中学到更好的学习。此外，MSRESNet和DNNET利用TRNR获得的性能比大多数最近的深度学习方法在大型标记数据集上受过训练的数据驱动。这些实验结果证实了所提出的TRNR的有效性和优越性。 TRNR的代码将很快公开。

人工智能和神经科学都深受互动。人工神经网络（ANNS）是一种多功能的工具，用于研究腹侧视觉流中的神经表现，以及神经科学中的知识返回激发了ANN模型，以提高任务的性能。但是，如何将这两个方向合并到统一模型中较少研究。这里，我们提出了一种混合模型，称为深度自动编码器，具有神经响应（DAE-NR），其将来自视觉皮质的信息包含在ANN中，以实现生物和人造神经元之间的更好的图像重建和更高的神经表示相似性。具体地，对小鼠脑和DAE-NR的输入相同的视觉刺激（即自然图像）。 DAE-NR共同学会通过映射函数将编码器网络的特定层映射到腹侧视觉流中的生物神经响应，并通过解码器重建视觉输入。我们的实验表明，如果只有在联合学习，DAE-NRS可以（i）可以提高图像重建的性能，并且（ii）增加生物神经元和人工神经元之间的代表性相似性。 DAE-NR提供了一种关于计算机视觉和视觉神经科学集成的新视角。